{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow tutorial\n",
    "_MILA, November 2017_\n",
    "\n",
    "## Protip: browsing through the TensorFlow API\n",
    "\n",
    "The [devdocs.io](http://devdocs.io/) website is an amazing resource to browse through the TensorFlow Python API (as well as many other APIs such as numpy or the Python API itself).\n",
    "\n",
    "# Using TensorFlow at MILA\n",
    "\n",
    "The most straightforward way to access TensorFlow using the MILA software stack is through the `tf1.4` conda environment. To activate the `tf1.4` environment, use the command\n",
    "\n",
    "```bash\n",
    "source activate tf1.4\n",
    "```\n",
    "\n",
    "To return back to normal, simply use the bash command\n",
    "\n",
    "```bash\n",
    "source deactivate\n",
    "```\n",
    "\n",
    "# Installing TensorFlow outside MILA\n",
    "\n",
    "Follow the [online documentation](https://www.tensorflow.org/install/), which describes how to install TensorFlow for all major platforms (Linux, macOS, Windows) in various ways (`virtualenv`, native `pip`, Docker, Anaconda).\n",
    "\n",
    "# Importing TensorFlow\n",
    "\n",
    "TensorFlow is imported as a Python package using the following statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Note: the `tensorflow` package is usually aliased to `tf` for convenience._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Paradigm\n",
    "\n",
    "TensorFlow separates the _definition_ of computation from its _execution_.\n",
    "\n",
    "Computation is defined via a [_dataflow_](https://en.wikipedia.org/wiki/Dataflow_programming) _graph_, i.e., a graph where nodes represent units of computation and the edges represent the data consumed or produced by the computation.\n",
    "\n",
    "TensorFlow calls these edges _tensors_ (not to be confused with the mathematical object of the same name). In TensorFlow parlance, a tensor is simply a multi-dimensional array of a certain data type.\n",
    "\n",
    "# Constant, variable, placeholder, and random tensors\n",
    "\n",
    "Many types of tensors may be used as input to the computation graph. We will cover four of them here: constant, variable, placeholder, and random tensors.\n",
    "\n",
    "## Constant\n",
    "\n",
    "A constant tensor always evaluates to the same value. It can be created using the `tf.constant` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"c:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "c = tf.constant(value=42.0, name='c')\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `name` argument is not strictly necessary, but it is considered good practice to name things in TensorFlow, as it facilitates visualizing the computation graph and debugging.\n",
    "\n",
    "To get the value associated with a constant tensor, we evaluate it within a session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value for c is 42.0\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    print('The value for c is {}'.format(session.run(c)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value for a constant _always_ stays the same, be it within the same session or across different sessions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value for c is 42.0\n",
      "The value for c is 42.0\n",
      "The value for c is 42.0\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    print('The value for c is {}'.format(session.run(c)))\n",
    "    print('The value for c is {}'.format(session.run(c)))\n",
    "\n",
    "with tf.Session() as session:\n",
    "    print('The value for c is {}'.format(session.run(c)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variable\n",
    "\n",
    "It can be useful for an input tensor's value to evolve across the lifetime of a session. For instance, a tensor's value can represent the weights of a neural network which we want to update using gradient descent.\n",
    "\n",
    "Tensors with this property are called _variables_. The preferred way to create variables is via `tf.get_variable`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'v:0' shape=(2,) dtype=float32_ref>\n"
     ]
    }
   ],
   "source": [
    "v = tf.get_variable(\n",
    "    name='v', shape=[2], dtype=tf.float32,\n",
    "    initializer=tf.zeros_initializer())\n",
    "print(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, the `name` argument is required. This is because TensorFlow refers to variables by name. As such, TensorFlow expects the name for the variable to be unique. Trying to create a variable with the same name will result in an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable v already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:\n"
     ]
    }
   ],
   "source": [
    "# Throughout this tutorial, we will wrap statements that\n",
    "# we know will cause an error to be raised with a try-except\n",
    "# block to print the error message only, and not the whole\n",
    "# stack trace.\n",
    "try:\n",
    "    tf.get_variable(name='v')\n",
    "except ValueError as e:\n",
    "    print(str(e).split('\\n')[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_(In case you are wondering, there is a way to bypass this behavior and retrieve by name a variable which has already been created. More on that later.)_\n",
    "\n",
    "Note that a variable's value only makes sense _within the context of a session_. Furthermore, a variable's initial value has to be set before it can be used. See what happens if we try to evaluate `v` within a session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attempting to use uninitialized value v\n",
      "\t [[Node: _retval_v_0_0 = _Retval[T=DT_FLOAT, index=0, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](v)]]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    try:\n",
    "        session.run(v)\n",
    "    except tf.errors.FailedPreconditionError as e:\n",
    "        print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow provides a function, `tf.global_variables_initializer`, which returns an op that can be evaluated to do just that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value for v is [ 0.  0.]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    session.run(tf.global_variables_initializer())\n",
    "    print('The value for v is {}'.format(session.run(v)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A variable's value persists across a session unless it is updated by running an assignment op. For instance, the op returned by `tf.assign_add` can be used to increment a variable's value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value for v is [ 0.  0.]\n",
      "The value for v is [ 0.  0.]\n",
      "The value for v is [ 1.  2.]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    session.run(tf.global_variables_initializer())\n",
    "    # The value for v persists across session.run calls...\n",
    "    print('The value for v is {}'.format(session.run(v)))\n",
    "    print('The value for v is {}'.format(session.run(v)))\n",
    "    # ... until it is updated by running an assignment op.\n",
    "    session.run(v.assign_add([1, 2]))\n",
    "    print('The value for v is {}'.format(session.run(v)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to `tf.assign_add`, the `tf.assign_sub` and `tf.assign` functions return ops which decrement and assign a variable's value, respectively.\n",
    "\n",
    "## Placeholder\n",
    "\n",
    "Oftentimes the computation we define depends on data which we don't yet have. For instance, the output of a neural network depends on a user-defined input which is only specified at runtime.\n",
    "\n",
    "_Placeholder_ tensors are used to represent this data. They can be created via `tf.placeholder`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"p:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "p = tf.placeholder(dtype=tf.float32, shape=[], name='p')\n",
    "print(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, the `name` argument is optional, but it is good practice to provide it.\n",
    "\n",
    "Because it has no pre-defined value, evaluating a placeholder tensor raises an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'p' with dtype float\n",
      "\t [[Node: p = Placeholder[dtype=DT_FLOAT, shape=[], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    try:\n",
    "        session.run(p)\n",
    "    except tf.errors.InvalidArgumentError as e:\n",
    "        # Cutting through the error message...\n",
    "        print('\\n'.join(str(e).split('\\n')[-3:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its value must be _explicitly_ passed to `session.run` via the `feed_dict` argument, which expects a `dict` mapping tensors to their value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value for p is 42.0\n",
      "The value for p is 21.0\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    # feed p the value 42\n",
    "    print('The value for p is {}'.format(session.run(p, feed_dict={p: 42})))\n",
    "    # feed p the value 21\n",
    "    print('The value for p is {}'.format(session.run(p, feed_dict={p: 21})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random\n",
    "\n",
    "Another useful input tensor to have in our toolbox is the random tensor. The random seed can be set globally via `tf.set_random_seed`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.set_random_seed(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many random distributions to choose from in TensorFlow. Let's look at `tf.random_uniform`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = tf.random_uniform(\n",
    "    shape=[], minval=0.0, maxval=1.0, dtype=tf.float32, name='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A random tensor's value changes randomly between `session.run` calls, but the sequence of those random values stays the same across different sessions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value for r is 0.8478444814682007\n",
      "The value for r is 0.23446130752563477\n",
      "The value for r is 0.8478444814682007\n",
      "The value for r is 0.23446130752563477\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    print('The value for r is {}'.format(session.run(r)))\n",
    "    print('The value for r is {}'.format(session.run(r)))\n",
    "\n",
    "with tf.Session() as session:\n",
    "    print('The value for r is {}'.format(session.run(r)))\n",
    "    print('The value for r is {}'.format(session.run(r)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combining tensors\n",
    "\n",
    "Tensors can be combined in various ways using what TensorFlow calls operations, or _ops_. Ops can take zero or more tensors as input and produce zero or more tensors as output, with or without side effects.\n",
    "\n",
    "We have already dealt with ops when initializing or assigning values to variables, but there are _many_ more TensorFlow functions which can be used to create ops.\n",
    "\n",
    "The best way to discover new useful ops is to browse the [TensorFlow Python API](https://www.tensorflow.org/api_docs/python/). For instance, we can discover that there exists a function, `tf.add`, which adds two tensors together and returns a tensor representing the output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 + 3 = 4\n"
     ]
    }
   ],
   "source": [
    "one_plus_three = tf.add(1, 3)\n",
    "\n",
    "with tf.Session() as session:\n",
    "    print('1 + 3 = {}'.format(session.run(one_plus_three)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that TensorFlow also offers syntactic sugar by overriding some Python operators like `+`, `-`, `*`, and `/`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 + 4 = 5\n"
     ]
    }
   ],
   "source": [
    "one_plus_four = tf.constant(1) + tf.constant(4)\n",
    "\n",
    "with tf.Session() as session:\n",
    "    print('1 + 4 = {}'.format(session.run(one_plus_four)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Where tensors live\n",
    "\n",
    "So far we created tensors and combined them together, but we have not explicitly dealt with the computation graph itself. Which computation graph are these tensors part of, then?\n",
    "\n",
    "By default, TensorFlow stores all tensors and operations in a **default graph**, which can be accessed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Operation 'c' type=Const>, <tf.Operation 'v/Initializer/zeros' type=Const>, <tf.Operation 'v' type=VariableV2>, <tf.Operation 'v/Assign' type=Assign>]\n"
     ]
    }
   ],
   "source": [
    "# Access the default graph\n",
    "default_graph = tf.get_default_graph()\n",
    "# Print first four operations defined in the graph\n",
    "print(default_graph.get_operations()[:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can change that default graph to be another graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default graph contains 23 operations, alternate graph contains 0 operations\n",
      "[<tf.Operation 'a' type=Const>, <tf.Operation 'b' type=Const>]\n",
      "default graph contains 23 operations, alternate graph contains 2 operations\n"
     ]
    }
   ],
   "source": [
    "# Create a new Graph\n",
    "alternate_graph = tf.Graph()\n",
    "\n",
    "# Print the number of operations defined in the default and alternate graphs\n",
    "print('default graph contains {} operations, '.format(len(default_graph.get_operations())) +\n",
    "      'alternate graph contains {} operations'.format(len(alternate_graph.get_operations())))\n",
    "\n",
    "# Use the alternate graph as the default graph\n",
    "with alternate_graph.as_default():\n",
    "    tf.constant(0, name='a')\n",
    "    tf.constant(1, name='b')\n",
    "    print(alternate_graph.get_operations())\n",
    "\n",
    "# Print again the number of operations defined in the default and alternate graphs\n",
    "print('default graph contains {} operations, '.format(len(default_graph.get_operations())) +\n",
    "      'alternate graph contains {} operations'.format(len(alternate_graph.get_operations())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, you usually won't have to deal with multiple computation graphs. We will however use multiple computation graphs in this tutorial to isolate ops in their own namespace where appropriate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow supports _automatic differentiation_, i.e., it can compute the derivative of scalars with respect to tensors in the graph and represent the result as a symbolic expression.\n",
    "\n",
    "Take for instance the linear equation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(dtype=tf.float32, shape=[], name='p')\n",
    "y = 3 * x + 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the derivative of `y` with respect to `x` with the `tf.gradients` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dy_dx, = tf.gradients(ys=y, xs=[x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can verify that the gradient evaluates to 3 as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The gradient of y with respect to x is 3.0\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    dy_dx_val = session.run(dy_dx)\n",
    "    print('The gradient of y with respect '\n",
    "          'to x is {}'.format(dy_dx_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Note: Some of you may have noticed that TensorFlow did not complain despite no value being provided for the `x` placeholder. This is because even though `x` is part of the computation graph, the derivative of `y` with respect to `x` does not involve `x`, and therefore evaluating it does not require a value to be passed for `x`._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "Find the minimum of the expression\n",
    "\n",
    "$$2(x - 2)^2 + 2(y + 3)^2$$\n",
    "\n",
    "using gradient descent by filling in the following code block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Avoid polluting the default graph by using an alternate graph\n",
    "with tf.Graph().as_default():\n",
    "    tf.set_random_seed(1234)\n",
    "\n",
    "    # Create two scalar variables, x and y, initialized at random.\n",
    "    # x = WRITEME.\n",
    "    # y = WRITEME.\n",
    "\n",
    "    # Create a tensor z whose value represents the expression\n",
    "    #     2(x - 2)^2 + 2(y + 3)^2\n",
    "    # z = WRITEME.\n",
    "    \n",
    "    # Compute the gradients of z with respect to x and y.\n",
    "    # dx, dy = WRITEME.\n",
    "    \n",
    "    # Create an assignment expression for x using the update rule\n",
    "    #    x <- x - 0.1 * dz/dx\n",
    "    # and do the same for y.\n",
    "    # x_update = WRITEME.\n",
    "    # y_update = WRITEME.\n",
    "    \n",
    "    with tf.Session() as session:\n",
    "        # Run the global initializer op for x and y.\n",
    "        # WRITEME.\n",
    "        \n",
    "        for _ in range(10):\n",
    "            pass\n",
    "            # Run the update ops for x and y.\n",
    "            # WRITEME.\n",
    "            \n",
    "            # Retrieve the values for x, y, and z, and print them.\n",
    "            # x_val, y_val, z_val = WRITEME.\n",
    "            # print('x = {:4.2f}, y = {:4.2f}, z = {:4.2f}'.format(x_val, y_val, z_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load tensorflow_exercise_solution.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimization made easy\n",
    "\n",
    "The solution to the exercise above can be shortened quite a bit by taking advantage of TensorFlow's optimization features. Here is the graph we were working with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.set_random_seed(1234)\n",
    "x = tf.get_variable(name='x', shape=[], dtype=tf.float32,\n",
    "                    initializer=tf.random_normal_initializer())\n",
    "y = tf.get_variable(name='y', shape=[], dtype=tf.float32,\n",
    "                    initializer=tf.random_normal_initializer())\n",
    "\n",
    "z = 2 * (x - 2) ** 2 + 2 * (y + 3) ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow provides utility classes to facilitate optimization in computation graphs. These classes inherit from `tf.train.Optimizer`. Let's look at the simplest one, `tf.train.GradientDescentOptimizer`.\n",
    "\n",
    "We instantiate the `tf.train.GradientOptimizer` by passing it a scalar learning rate. Note that the learning rate itself can be symbolic, and is allowed to vary across a session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then call the optimizer's `minimize` function to obtain an op with, when evaluated, does a gradient descent step on the variables we specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "update_op = optimizer.minimize(loss=z, var_list=tf.trainable_variables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we took advantage of the fact that all variables created via `tf.get_variable` can be accessed as a list using the `tf.trainable_variables` function.\n",
    "\n",
    "(Note: you can pass `trainable=False` to `tf.get_variable` to exclude a certain variable from ending up in that list.)\n",
    "\n",
    "The code then proceeds as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = 0.52, y = -1.75, z = 7.49\n",
      "x = 1.11, y = -2.25, z = 2.70\n",
      "x = 1.47, y = -2.55, z = 0.97\n",
      "x = 1.68, y = -2.73, z = 0.35\n",
      "x = 1.81, y = -2.84, z = 0.13\n",
      "x = 1.89, y = -2.90, z = 0.05\n",
      "x = 1.93, y = -2.94, z = 0.02\n",
      "x = 1.96, y = -2.97, z = 0.01\n",
      "x = 1.98, y = -2.98, z = 0.00\n",
      "x = 1.99, y = -2.99, z = 0.00\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as session:\n",
    "    session.run(tf.global_variables_initializer())\n",
    "\n",
    "    for _ in range(10):\n",
    "        session.run(update_op)\n",
    "        x_val, y_val, z_val = session.run([x, y, z])\n",
    "        print('x = {:4.2f}, y = {:4.2f}, z = {:4.2f}'.format(x_val, y_val, z_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Control flow\n",
    "\n",
    "TensorFlow implements symbolic operations for executing common control flow structures, like if-statements and while loops.\n",
    "\n",
    "This is usually where people get confused: despite appearances, there is a *big* difference between a regular control flow statement and its symbolic counterpart. **When working with control flow ops, you should always keep in mind that the code you write _defines the computation graph_, it does not _execute_ that computation.**\n",
    "\n",
    "## If-statement\n",
    "\n",
    "To illustrate this, let's look at a regular if-statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "regular_if_graph = tf.Graph()\n",
    "with regular_if_graph.as_default():\n",
    "    # Define computation graph\n",
    "    x = 3\n",
    "    y = tf.placeholder(dtype=tf.float32, shape=[], name='y')\n",
    "    if x < 4:\n",
    "        z = y + 1\n",
    "    else:\n",
    "        z = y - 1\n",
    "    \n",
    "    # Run the computation graph on some input\n",
    "    with tf.Session() as session:\n",
    "        print(session.run(z, feed_dict={y: 0}))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This piece of code is fairly simple: depending on the value of `x`, we either add 1 to `y` or subtract 1 from it. However, it depends on the fact that *the value of `x` was known when the graph was created*. In fact, because the `else` statement never gets executed, **the expression `y - 1` doesn't even appear in the computation graph**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Operation 'y' type=Placeholder>, <tf.Operation 'add/y' type=Const>, <tf.Operation 'add' type=Add>]\n"
     ]
    }
   ],
   "source": [
    "print(regular_if_graph.get_operations())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, this approach does not work when the value of `x` is not known in advance. Try the same code with a placeholder `x` and you will be greeted with an error message:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using a `tf.Tensor` as a Python `bool` is not allowed. Use `if t is not None:` instead of `if t:` to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    x = tf.placeholder(dtype=tf.float32, shape=[], name='x')\n",
    "    y = tf.placeholder(dtype=tf.float32, shape=[], name='y')\n",
    "    try:\n",
    "        if x < 4:\n",
    "            z = y + 1\n",
    "        else:\n",
    "            z = y - 1\n",
    "    except TypeError as e:\n",
    "        print(str(e).split('\\n')[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this use case, TensorFlow implements function called `tf.cond` which acts as a symbolic counterpart to the if-statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "-1.0\n"
     ]
    }
   ],
   "source": [
    "symbolic_if_graph = tf.Graph()\n",
    "with symbolic_if_graph.as_default():\n",
    "    # Define computation graph\n",
    "    x = tf.placeholder(dtype=tf.float32, shape=[], name='x')\n",
    "    y = tf.placeholder(dtype=tf.float32, shape=[], name='y')\n",
    "    z = tf.cond(\n",
    "        pred=x < 4,\n",
    "        true_fn=lambda: y + 1,\n",
    "        false_fn=lambda: y - 1)\n",
    "    \n",
    "    # Run the computation graph on some inputs\n",
    "    with tf.Session() as session:\n",
    "        print(session.run(z, feed_dict={x: 3, y: 0}))\n",
    "        print(session.run(z, feed_dict={x: 5, y: 0}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.cond` function takes a predicate `pred`, a subgraph-creating function `true_fn`, and a subgraph-creating function `false_fn` as input. The predicate is a *symbolic* boolean which is used to decide which branch of the conditional is executed. The two graph-creating functions take no argument as input, create a computation subgraph, and return its symbolic output.\n",
    "\n",
    "Looking at the operations defined in the graph above reveals a very different picture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Operation 'x' type=Placeholder>, <tf.Operation 'y' type=Placeholder>, <tf.Operation 'Less/y' type=Const>, <tf.Operation 'Less' type=Less>, <tf.Operation 'cond/Switch' type=Switch>, <tf.Operation 'cond/switch_t' type=Identity>, <tf.Operation 'cond/switch_f' type=Identity>, <tf.Operation 'cond/pred_id' type=Identity>, <tf.Operation 'cond/add/y' type=Const>, <tf.Operation 'cond/add/Switch' type=Switch>, <tf.Operation 'cond/add' type=Add>, <tf.Operation 'cond/sub/y' type=Const>, <tf.Operation 'cond/sub/Switch' type=Switch>, <tf.Operation 'cond/sub' type=Sub>, <tf.Operation 'cond/Merge' type=Merge>]\n"
     ]
    }
   ],
   "source": [
    "print(symbolic_if_graph.get_operations())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The takeaway here is that because TensorFlow separates graph definition from graph execution, we have to adjust our mental model of what happens behind the scenes, even for a seemingly innocuous if-statement.**\n",
    "\n",
    "Now that this warning is out of the way, we can look at some other symbolic control-flow functions TensorFlow implements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case\n",
    "\n",
    "The `tf.case` function implements a symbolic counterpart to the `case` control flow statement. It takes a sequence of (predicate, subgraph-creating function) tuples as input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n",
      "1.0\n",
      "4.0\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    # Define computation graph\n",
    "    x = tf.placeholder(dtype=tf.int32, shape=[], name='x')\n",
    "    y = tf.placeholder(dtype=tf.float32, shape=[], name='y')\n",
    "    z = tf.case(\n",
    "        pred_fn_pairs=[(tf.equal(x, 0), lambda: y + 1),\n",
    "                       (tf.equal(x, 1), lambda: y - 1),\n",
    "                       (tf.equal(x, 2), lambda: y * 2)],\n",
    "        default=lambda: y / 2)\n",
    "    \n",
    "    # Run the computation graph on some inputs\n",
    "    with tf.Session() as session:\n",
    "        print(session.run(z, feed_dict={x: 0, y: 2}))\n",
    "        print(session.run(z, feed_dict={x: 1, y: 2}))\n",
    "        print(session.run(z, feed_dict={x: 2, y: 2}))\n",
    "        print(session.run(z, feed_dict={x: 3, y: 2}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## While loop\n",
    "\n",
    "The `tf.while_loop` function implements a symbolic counterpart to the `while` control flow statement. It takes a `cond` subgraph-creating function, a `body` subgraph-creating function and a `loop_vars` sequence of tensors.\n",
    "\n",
    "You can think of `loop_vars` as the initial state of all tensors which change from one iteration of the loop to the other. The `cond` and `body` functions takes a sequence of tensors with the same length as `loop_vars` as input; you can think of them as the current state the `loop_vars`. The `cond` function returns a symbolic boolean telling whether the loop should be executed or not. The `body` function returns a sequence of tensors representing the new values the `loop_vars`.\n",
    "\n",
    "Here is an example of a sequential implementation of the Fibonacci sequence using `tf.while_loop`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "13\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    # Define computation graph\n",
    "    n = tf.placeholder(tf.int32, shape=[], name='n')\n",
    "    i = tf.constant(2)\n",
    "    a = tf.constant(0)\n",
    "    b = tf.constant(1)\n",
    "\n",
    "    _, _, nth_fib = tf.while_loop(\n",
    "        cond=lambda i, a, b: i < n,\n",
    "        body=lambda i, a, b: (i + 1, b, a + b),\n",
    "        loop_vars=(i, a, b))\n",
    "\n",
    "    # Run the computation graph on some inputs\n",
    "    with tf.Session() as session:\n",
    "        print(session.run(nth_fib, feed_dict={n: 6}))\n",
    "        print(session.run(nth_fib, feed_dict={n: 8}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a placeholder tensor `n` representing which element of the Fibonacci sequence to compute. We then instantiate three constant tensors that act as the `loop_vars`: a counter `i` and two values `a` and `b` representing the elements $i - 1$ and $i$ of the Fibonacci sequence, respectively.\n",
    "\n",
    "The `cond` argument passed to `tf.while_loop` indicates that the body should execute as long as $i < n$. The `body` argument itself increments `i` by one and updates the value for `a` and `b` so they reflect elements $i$ and $i + 1$ of the Fibonacci sequence, respectively (reminder: $fib(i + 1) = fib(i) + fib(i - 1)$).\n",
    "\n",
    "## Scan\n",
    "\n",
    "Another useful function TensorFlow implements is `tf.scan`, which \"scans\" over the elements of its input and applies some possibly stateful function to them.\n",
    "\n",
    "Here is an example of a cumulative sum implemented using `tf.scan`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1  3  6 10]\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    # Define computation graph\n",
    "    x = tf.placeholder(tf.int32, shape=[None], name='x')\n",
    "\n",
    "    c = tf.scan(\n",
    "        fn=lambda a, x_t: a + x_t,\n",
    "        elems=x,\n",
    "        initializer=tf.constant(0))\n",
    "    \n",
    "    # Run the computation graph on some input\n",
    "    with tf.Session() as session:\n",
    "        print(session.run(c, feed_dict={x: [1, 2, 3, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pass `tf.scan` three arguments as inputs: a subgraph-creating function `fn`, a tensor (or sequence of tensors) `elems` to loop over, and a tensor (or list of tensors) `initializer` of initial accumulator values.\n",
    "\n",
    "The scan function loops over the first axis of `elems` (or, if it's a sequence, over the first axes of every element in the sequence). The `fn` function takes the accumulator value and the current loop element as input, and returns a new value for the accumulator.\n",
    "\n",
    "The `tf.scan` function itself returns a sequence of accumulated values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scaling up to large computation graphs\n",
    "\n",
    "We have now covered the bare minimum that would allow you to do machine learning with TensorFlow. We have not covered _every_ TensorFlow op, but you now possess the knowledge required to browse through the [TensorFlow Python API](https://www.tensorflow.org/api_docs/python/) and find what you need.\n",
    "\n",
    "We will now concentrate on ways to scale what you learned to actual machine learning problems without increasing the maintenance complexity too much.\n",
    "\n",
    "## Variable and name scopes\n",
    "\n",
    "TensorFlow uses a soft convention for op and variable names: an op or variable that is part of a hierarchy should have a name that conveys its location in the hierarchy, with the `'/'` character being used to separate different levels in the hierarchy. For instance, a good name for the bias vector of the second layer of the model would be `'model/layer2/b'`.\n",
    "\n",
    "In order to reduce code duplication and facilitate maintenance, TensorFlow provides two context managers, named `tf.name_scope` and `tf.variable_scope`, inside which variables and ops that are created see their name prepended with the name of the enclosing scope. The difference between the two is that `tf.variable_scope` operates on _all_ names, whereas `tf.name_scope` operates on all _but_ variable names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "foo/bar/a:0\n",
      "foo/bar/b:0\n",
      "a:0\n",
      "machine/learning/b:0\n"
     ]
    }
   ],
   "source": [
    "# Variable scopes operate on all tensors\n",
    "with tf.variable_scope('foo'):\n",
    "    # Scopes can be nested\n",
    "    with tf.variable_scope('bar'):\n",
    "        print(tf.get_variable('a', shape=[]).name)\n",
    "        print(tf.constant(0.0, name='b').name)\n",
    "# Name scopes do not operate on variables\n",
    "with tf.name_scope('machine'):\n",
    "    with tf.name_scope('learning'):\n",
    "        print(tf.get_variable('a', shape=[]).name)\n",
    "        print(tf.constant(0.0, name='b').name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Device placement\n",
    "\n",
    "In this tutorial we have not bothered with the specific placement (CPU or GPU) of our ops, i.e., the device on which they are executed, mostly because the examples we considered were so small in scale that GPU acceleration makes little to no sense.\n",
    "\n",
    "However, in general, we would like most of our large operations (such as matrix-matrix multiplications) to take place on the GPU. Fortunately for us, TensorFlow already handles device placement for us behind the scenes. The short story is that TensorFlow will try to place all possible ops on all available GPUs, which is a good default to have.\n",
    "\n",
    "Because we are sharing workstations with multiple GPUs, this means we need to be careful in allowing TensorFlow to see only the GPUs we want to use. This is achieved by setting the `CUDA_VISIBLE_DEVICES` environment variable. For instance, to allow TensorFlow to see GPUs 0 and 2, set it to\n",
    "\n",
    "```bash\n",
    "CUDA_VISIBLE_DEVICES=0,2\n",
    "```\n",
    "\n",
    "To allow TensorFlow to see GPU 0 only, set it to\n",
    "\n",
    "```bash\n",
    "CUDA_VISIBLE_DEVICES=0\n",
    "```\n",
    "\n",
    "To disallow it to see any GPU, set it to\n",
    "\n",
    "```bash\n",
    "CUDA_VISIBLE_DEVICES=\"\"\n",
    "```\n",
    "\n",
    "In addition, there may be situations in which we want fine-grained control over device placement. For instance, we may want on-the-fly input pre-processing to be computed on the CPU and reserve the GPU for inference. In that case, use the `tf.device` context manager:\n",
    "\n",
    "```python\n",
    "# All ops created within this context are forced to be placed on CPU\n",
    "with tf.device('/cpu:0'):\n",
    "    x = # some tensor\n",
    "    preprocessed_x = # some pre-processing on x\n",
    "\n",
    "# All ops created within this context are forced to be placed on GPU\n",
    "with tf.device('/gpu:0'):\n",
    "    y = # some mapping from x to y\n",
    "```\n",
    "\n",
    "You can find more information on manual device placement in the [TensorFlow documentation](https://www.tensorflow.org/tutorials/using_gpu#manual_device_placement)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced topics\n",
    "\n",
    "This tutorial does not cover all topics in the [TensorFlow programmer's guide](https://www.tensorflow.org/programmers_guide/). The guide as a whole is a great follow-up read; in particular, you will find the following sections useful:\n",
    "\n",
    "* [Data management](https://www.tensorflow.org/programmers_guide/datasets)\n",
    "* [Estimators](https://www.tensorflow.org/programmers_guide/estimators)\n",
    "* [Eager mode](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager)\n",
    "\n",
    "You may also want to have a look at the [performance guide](https://www.tensorflow.org/performance/performance_guide), as well as the tutorial series on TensorBoard:\n",
    "\n",
    "* [Visualizing learning](https://www.tensorflow.org/get_started/summaries_and_tensorboard)\n",
    "* [Graph visualization](https://www.tensorflow.org/get_started/graph_viz)\n",
    "* [Histogram dashboard](https://www.tensorflow.org/get_started/tensorboard_histograms)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
